Protocol for the Evaluation of Data-Related Processes and Challenges at Level 1–4 Trauma Centers in Arkansas: A Mixed-Methods Case Study

Background: In the following manuscript, we describe the detailed protocol for a mixed-methods, observational case study conducted to identify and evaluate existing data-related processes and challenges currently faced by trauma centers in a rural state. The data will be utilized to assess the impact of these challenges on registry data collection. Methods: The study relies on a series of interviews and observations to collect data from trauma registry staff at level 1–4 trauma centers across the state of Arkansas. A think-aloud protocol will be used to facilitate observations as a means to gather keystroke-level modeling data and insight into site processes and workflows for collecting and submitting data to the Arkansas Trauma Registry. Informal, semi-structured interviews will follow the observation period to assess the participant’s perspective on current processes, potential barriers to data collection or submission to the registry, and recommendations for improvement. Each session will be recorded and de-identified transcripts and session notes will be used for analysis. Keystroke level modeling data derived from observations will be extracted and analyzed quantitatively to determine time spent performing end-to-end registry-related activities. Qualitative data from interviews will be reviewed and coded by 2 independent reviewers following a thematic analysis methodology. Each set of codes will then be adjudicated by the reviewers using a consensus-driven approach to extrapolate the final set of themes. Discussion: We will utilize a mixed methods approach to understand existing processes and barriers to data collection for the Arkansas Trauma Registry. Anticipated results will provide a baseline measure of the data collection and submission processes at various trauma centers across the state. We aim to assess strengths and limitations of existing processes and identify existing barriers to interoperability. These results will provide first-hand knowledge on existing practices for the trauma registry use case and will provide quantifiable data that can be utilized in future research to measure outcomes of future process improvement efforts. The potential implications of this study can form the basis for identifying potential solutions for streamlining data collection, exchange, and utilization of trauma registry data for clinical practice, public health, and clinical and translational research.


Background
Registries play a critical role in clinical practice, research, and public health by providing a rich source of healthcare information that can be leveraged to improve patient and population health outcomes (1).The data collected to support registries are derived from local, state, and national sources, providing a window into healthcare at all levels.As more federal regulations require reporting to clinical data registries, the role of registries in clinical practice, research, and public health has never been more signi cant.In addition to being a rich data source for clinical research and public health surveillance, these specialized data sets are often relied on for use in routine clinical practice, which require data to be timely and of high quality.
A trauma registry is a data registry or repository with the explicit purpose of data collection of uniform, clinical data elements that "document acute care delivered to patients hospitalised with injuries" (2).In general, registry data can be used to support clinical practice (e.g., care coordination, performance improvement), clinical and public health research, and public health initiatives (e.g., surveillance, epidemiological and injury prevention) (3).Trauma registries are designed to "improve the e ciency and quality of trauma care" (2) and, at regional and national levels, trauma registries can offer large datasets that can be used for comprehensive public health surveillance of injuries (4) and to evaluate patient outcomes within and across facilities (2).
Trauma centers across the US are identi ed and designated by the American Trauma Society (5) into 5 different levels (levels 1-5), after completing designation and veri cation processes.Level determination is dependent on the kind of resources available at the center and the number of patients admitted yearly (6).This provides a national standard for trauma care in hospitals.Level 1 refers to a comprehensive regional resource that is a tertiary care facility central to the trauma system (e.g., Cedars-Sinai Medical Center, Massachusetts General Hospital) (5).In contrast, level 5 trauma centers provide only the initial evaluation and stabilization, have limited diagnostic capabilities, and most often prepare patients for transfer to higher levels of care (5).

Arkansas Trauma Registry
The Arkansas Trauma Registry (ATR) (3) is a statewide trauma data collection and evaluation system composed of standardized data elements that include details of each injury event, patient demographics, prehospital information, diagnoses, in-hospital care, and eventual outcomes of the injured patients admitted to facilities across the rural state of Arkansas.The goal of the ATR (and registries in general) is to obtain, code, and maintain the data for analysis and eventual reporting of individual and aggregate results.Arkansas has level 1-4 trauma centers [1] , all of which participate in the ATR by providing a standardized set of real-world data [2] from the electronic health record (EHR) on various trauma patients (referred to as cases).Sites across the state of Arkansas also participate in the National Trauma Data Bank ® (NTDB ® ) Trauma Program, which is the largest aggregation of US trauma registry data available for research and reporting (e.g., benchmarking and performance improvement) (7).
Sites participating in the ATR (and/or NTDB ® ) are required to submit data on an ongoing basis and invest a substantial amount of time and effort (not to mention, money) on collecting, coding, performing quality reviews of, and submitting data.In most cases, these processeswhich are not unique to trauma registriesare highly manual and burdensome (2,4,8).Registries often rely on manual medical record abstraction (or chart review) to gather data from EHRs and emergency medical services (EMS) systems.Such manual approaches are expensive and, often, ine cient, primarily due to the complexities associated with data collection or abstraction and mapping of the data to t the registry data model (9).Data Quality and Process Improvement Medical information is complex.Designing a data collection process that captures its intricacies while smoothly tting into the work ow is especially important.Because trauma registrars, registry coordinators, and program managers are directly involved in the data collection process, they have a good understanding of the procedures, work ows, and variables contributing to the overall quality of data that is submitted; and data collection can be improved with their guidance.Trauma registries need data quality improvements in multiple areas; most notably, standardization and missing data are common areas of weakness impacted by the collection process.A recent study found that 70% of trauma centers were missing comorbidity data (10).Historically, data quality monitoring and management have been inconsistent, but studies show that con dence in the registry data is directly related to data quality (11).
In fact, periodic data quality reviews should be conducted using systematic processes to track and improve these issues (12).In general, data standardization must underlie the process of data acquisition and exchange to support generalized trauma care quality improvements.Curtis and colleagues (13) report that a standardized approach to reviewing trauma cases across hospitals is a priority for trauma quality improvement.Advances in systems interoperability, i.e., through the use of standards-based methods and direct EHR data exchange, have the potential to address these issues by automating part or all of the data collection and submission processes.
In the following manuscript, we describe the detailed protocol for a mixed-methods, observational case study conducted by researchers at the University of Arkansas for Medical Sciences (UAMS).This study incorporates the perspectives of experienced trauma personnel to identify aspects of registry submission that could be made more e cient.The results of this research has the potential to save money and enhance the quality of the data obtained, leading to improved performance and trauma standards of care, and, ultimately, better patient outcomes.

Objective
The primary objective of this study is to evaluate existing data collection and exchange processes of trauma centers in a rural state (Arkansas) to identify data-related challenges and the potential impact on registry data collection and submission to a state-based clinical data registry (Arkansas Trauma Registry).A secondary objective is to calculate the average time spent on data collection and submission by generating work ow and process-level metrics using keystroke-level modeling (KLM) (14).

Study Design
This study employs a mixed-methods, observational case study design for data collection and analysis through the use of semi-structured observations and interviews (Table 1).A detailed investigation will be conducted of the current data collection processes for the ATR across level 1-4 trauma centers within the state of Arkansas.We will rely on direct observations and interviews as our primary data sources to (1) gather qualitative information from registry personnel across unique sites on current processes, challenges, and perceptions (primary objective) of the end-to-end data-related practices carried out at each site; (2) generate data-and work ow diagrams corresponding to the processes outlined by the site (primary objective); and (3) collect KLM data (quantitative) based on observations and verbal walkthroughs (secondary objective) using a think-aloud protocol.This study has received a determination of not human subjects research as de ned in 45 CFR 46.102 by the UAMS Institutional Review Board.Objective: To evaluate existing data-related processes and challenges currently faced by Arkansas trauma centers submitting data to the ATR.
Study Design: Mixed-Methods, Observational Case Study Data Collection: A 30-minute observation period detailing walkthrough of typical data collection and submission to the ATR (to collect quantitative, KLM data), followed by a 30-minute informal interview describing challenges to data collection and submission to the ATR (to collect qualitative data).
Analysis: Thematic analysis of interview data (qualitative) to identify common themes as to the perceptions of current data-centric processes at sites.KLM of observation data (quantitative) to generate baseline time estimates for data collection and submission at each site.

Study Participants
ATR personnel employed at level 1-4 trauma centers across the state of Arkansas will be recruited to participate in this study.Participants will be included if they are clinical, research, or data registry personnel who actively participate in the day-to-day data collection and submission processes for the ATR.More speci cally, we are interested in the perspectives of trauma program managers or supervisors, registrars, coordinators, and data entry staff (all roles hereinafter referred to as registrars) responsible for capturing, entering, tracking, and submitting patient data to the ATR.There are no exclusion criteria for this study.
We will rely on a combination of purposive (15) and snowball (16, 17) sampling to identify and recruit participants from Arkansas trauma centers throughout the state.Purposive sampling is a commonly used sampling method in qualitative research (15) to select "information-rich cases for the most effective use of limited resources."We will rst identify trauma centers in Arkansas through publicly available information from the Arkansas Department of Health.Once sites have been identi ed, we will identify key personnel from each site for inclusion, engaging with the Arkansas Department of Health for assistance, where appropriate.
Participation is voluntary and informed consent to participate will be obtained.Emails will be sent to potential participants brie y outlining the objectives of the study and details of participation.In addition, contact information for the primary investigator (MYG) will be included to allow participants to follow up with questions or request additional information before committing to participate in the study.Those expressing interest will be scheduled for an observation and an interview session.In situations where a potential participant does not respond to the initial email, 3 additional follow-up emails will be attempted before ceasing recruitment attempts.Participation is completely voluntary.By participating in the observations and interviews, the participants are acknowledging informed consent, and the responses to questions and discussion points during each session will be considered part of the dataset collected for the study and will be included in the analysis.
At minimum, we aim to include personnel from at least 4 sites, each site representing a different trauma certi cation level.This will ensure we are able to identify potential differences between sites of different levels.From each site, we aim to include at least 1 person but will attempt to interview all that are available and willing to participate.It is likely that sites will vary in the number of relevant personnel, assuming anywhere between 1 and 5 personnel per site.Therefore, we look to recruit between 4 (1 person per site across 4 sites) to 20 (5 persons per site across 4 sites) total participants.However, recruitment will continue only until we have reached data saturation (18).In other words, as long as we reach our threshold of 1 person across 4 unique sites of varying trauma levels, we will stop recruitment if we establish that data collected during observations and interviews is repetitive and no new data is being expressed.
Compensation.Participants will be eligible for compensation upon completion of study activities (one observation and one interview session).Participants will be eligible to receive a $50 gift card for completing one, 30-minute observation session, and another $50 gift card for completing one, 30-minute interview session, for a maximum of $100.
Risks and Bene ts.This research is an observational study that relies on interviews, observations, and surveys.The probability and magnitude of harm or discomfort anticipated in the research are not greater than those ordinarily encountered in daily life or during the performance of routine physical and psychological examinations or tests and where con dentiality is properly protected.There are no physical risks associated with this study.However, there is a potential loss of con dentiality risk for participants.Measures to protect the con dentiality of study data will be implemented, and every effort will be made to keep their information con dential; however, this cannot be guaranteed.For this reason, participants will be advised of this limitation during the informed consent process.Other than the compensation for participation, there is no direct bene t for participation in this study.However, knowledge gained from the study could potentially bene t patients in the future.We hope that the information gained will increase our knowledge of which sites and clinical studies are most appropriate for standards-based, data exchange technologies.

Study Setting
This study will be conducted online (virtually) using the Zoom Video Conferencing platform (https://zoom.us/)licensed by UAMS.Virtual sessions will be closed and available by invitation only.Observations and interviews will be recorded (video and audio) for analysis.Registry personnel can join the virtual meeting room and opt-in to share their camera and computer screen(s) to display any relevant information during the session.All research activities will be conducted by appropriately quali ed members of the research team in a discrete setting.Trained members of the research team will carry out the virtual sessions in a private room, such as a work or home o ce with a closed door and away from others, to maintain a private environment.Participants can join from their typical work environment (e.g., work or home o ce setting) and have full control over the setting from which they join.However, for the observations in particular, it is optimal that the participant joins from their primary work environment to facilitate accurate work ow with on-site technologies, including program layout and keystroke maneuvers.
We will allocate 30 minutes each for observation and interview sessions, for a total time commitment of 60 minutes.To accommodate participant schedules, the sessions can occur concurrently, as a single, 60minute meeting, or be scheduled separately, as two distinct 30-minute sessions.In the event that the allocated time is not su cient to complete either session, the study team will invite the participant to one additional 30-minute session to complete any remaining components of the observation and/or interview.
Observations and interviews will follow a semi-structured format, referencing an interview guide developed prior to study start.The observation/interview guide(s) will include a script with example prompts for guidance to help drive the conversation and encourage the interviewee to describe processes and opinions in as much detail as possible.The interview guide will be a working (or living) document that can be modi ed or adapted after each interview, to allow the research team to learn and gather new insight that will inform future interviews.
Observations and interviews will be proctored by a trained member of the research team who will be responsible for managing the meeting, relying on the interview guide for guidance.Responsibilities will include managing and maintaining the recording during the session, following up with prompts or additional, clarifying questions (where appropriate), capturing eld notes (as needed), and closing out the session, which may or may not require scheduling a follow-up session.A second member of the research team may join the session as a technical backup and to assist with note-taking.

Data Collection
We will use observations and interviews to collect data from personnel at Arkansas trauma centers that are involved in the end-to-end data collection and submission to the ATR. Figure 1 provides an overview of the data collection processes to be followed for this study.Observations provide an avenue for participants to walk through current site-level processes with the research team.The walkthroughs follow a think-aloud protocol (19,20), which will be used to allow the evaluator to assess the ease, challenges, and comprehension of the tasks being performed by the endusers (e.g., registrars collecting and submitting data to the ATR).Think-aloud protocols allow the interviewee to walk through the entire process from beginning to end, while verbally explaining steps aloud as they complete each task.Think-aloud facilitates a thorough description of existing processes without introducing potential biases of the researcher.For example, a participant describing how to exit an internet browser window may say, "I have completed my task and am ready to close out of the system.To do so, I am placing my hand on the mouse, moving the cursor to the right-hand corner of the browser window, where I notice a red 'X', and then, I single-click the red 'X' to close the window."Each step will be analyzed and quantitative data extracted for the KLM analysis.Referring back to the example stated above, we can derive key operators and predict execution times as follows: the mental preparation and acknowledgement of a needed action ("ready to close out of the system"), assigned operator "M", would take about 1.35 seconds; placing the hand on the mouse, operator "H", would take about 0.40 seconds; movement of the mouse and pointing to the target, operator "P", would take about 1.10 seconds; and the nal "single click on the red 'X' to close the window", operator "K", would take about 0.20 seconds.These times would then be summed to get the overall time for the activity (t = 3.05 seconds).
In addition to generating the KLM results, data gathered during the course of the observation will be used to create data-and work ow diagrams, which will be reviewed by the interviewee(s) to ensure a common understanding of existing processes.Data ow diagrams will be generated to depict the end-to-end movement of the data from the moment a trauma case is rst identi ed, to when the nal set of data for a case is submitted to the registry.Work ow diagrams will be generated to complement data ow, and will depict how the site staff moves through the data-related processes from start to nish.
The semi-structured interviews, proctored by a trained member of the research team, are informal and conversational.Although the research team has developed several general questions as prompts for discussion, participants may express concerns or obstacles that are not adequately addressed by the standard prompts, so interviewers may expand with open-ended follow-up questions, as appropriate.Encouraging the interviewee to guide discussion creates exibility during the interview that ensures attention to and appreciation for critical and unique processes at each site.In general, interviews are centered on the day-to-day activities and data-related tasks of each interviewee as it relates to their work with the ATR.Speci c topics of interest include: standard process/work ow; typical work/case load on a daily, weekly, and/or monthly basis; typical data collection procedures; perceived issues or barriers to collection; perceived overall data quality; recommendations or suggestions for process improvement.No personal information will be collected.

Data Analysis
Qualitative from the observations and interviews will be analyzed using a constant comparative approach (21), which focuses on illuminating barriers and facilitators in technological, procedural, and personnel attributes as it relates to current data collection procedures.Figure 2 depicts the process ow from data collection to nal analysis and results.
Through thematic analysis (Fig. 3), we aim to identify common themes related to processes, barriers, and time metrics.Using a Rapid Qualitative Evaluation approach (22)(23)(24), all qualitative data will be coded,

Table 1
Protocol OverviewContext: Level 1-4 trauma centers within the state of Arkansas.